{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units\n",
    "### Authors: Zihao Zhang and Stefan Zohren\n",
    "### Oxford-Man Institute of Quantitative Finance, Department of Engineering Science, University of Oxford\n",
    "\n",
    "This jupyter notebook is used to demonstrate our recent paper [2]. We use FI-2010 [1] dataset and present how model architecture is constructed here. The FI-2010 is publicly avilable and interested readers can check out their paper [1]. The dataset can be downloaded from: https://etsin.fairdata.fi/dataset/73eb48d7-4dbc-4a10-a52a-da745b47a649\n",
    "\n",
    "Otherwise, it can be obtained from: https://drive.google.com/drive/folders/1Xen3aRid9ZZhFqJRgEMyETNazk02cNmv?usp=sharing\n",
    "\n",
    "[1] Ntakaris A, Magris M, Kanniainen J, Gabbouj M, Iosifidis A. Benchmark dataset for mid‐price forecasting of limit order book data with machine learning methods. Journal of Forecasting. 2018 Dec;37(8):852-66. https://arxiv.org/abs/1705.03233\n",
    "\n",
    "[2] Zhang Z, Zohren S. Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units. https://arxiv.org/abs/2105.10430\n",
    "\n",
    "#### This notebook demonstrates how to train DeepLOB-Attention by using tensorflow 2 on IPUs.\n",
    "\n",
    "#### For more information about IPU, please check https://www.graphcore.ai/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import os\n",
    "import logging\n",
    "import glob\n",
    "import argparse\n",
    "import sys\n",
    "import time\n",
    "import tensorflow as tf\n",
    "from tensorflow.python import ipu\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "\n",
    "# load my packages\n",
    "from preprocess import *\n",
    "from model import get_model_seq, get_model_attention"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# please change the data_path to your local path\n",
    "data_path = '/home/zihaoz/deeplob/data'\n",
    "\n",
    "T = 50 # lookback window size\n",
    "epochs = 150 # number of training epochs\n",
    "batch_size = 16 # gradient descent batch size\n",
    "n_hidden = 64 # hidden state for decoder\n",
    "SHUFFLE=True # shuffle the traning data\n",
    "saved_model_path = './model_deeplob_attention/deeplob_attention'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_encoder_input.shape = (254701, 50, 40, 1),train_decoder_target.shape = (254701, 5, 3)\n",
      "test_encoder_input.shape = (139538, 50, 40, 1),test_decoder_target.shape = (139538, 5, 3)\n"
     ]
    }
   ],
   "source": [
    "# load data\n",
    "dec_train = np.loadtxt(data_path + '/Train_Dst_NoAuction_DecPre_CF_7.txt')\n",
    "dec_test1 = np.loadtxt(data_path + '/Test_Dst_NoAuction_DecPre_CF_7.txt')\n",
    "dec_test2 = np.loadtxt(data_path + '/Test_Dst_NoAuction_DecPre_CF_8.txt')\n",
    "dec_test3 = np.loadtxt(data_path + '/Test_Dst_NoAuction_DecPre_CF_9.txt')\n",
    "dec_test = np.hstack((dec_test1, dec_test2, dec_test3))\n",
    "\n",
    "# extract limit order book data from the FI-2010 dataset\n",
    "train_lob = prepare_x(dec_train)\n",
    "test_lob = prepare_x(dec_test)\n",
    "\n",
    "# extract label from the FI-2010 dataset\n",
    "train_label = get_label(dec_train)\n",
    "test_label = get_label(dec_test)\n",
    "\n",
    "# prepare training data. We feed past T observations into our algorithms.\n",
    "train_encoder_input, train_decoder_target = data_classification(train_lob, train_label, T)\n",
    "train_decoder_input = prepare_decoder_input(train_encoder_input, teacher_forcing=False)\n",
    "\n",
    "test_encoder_input, test_decoder_target = data_classification(test_lob, test_label, T)\n",
    "test_decoder_input = prepare_decoder_input(test_encoder_input, teacher_forcing=False)\n",
    "\n",
    "print(f'train_encoder_input.shape = {train_encoder_input.shape},'\n",
    "      f'train_decoder_target.shape = {train_decoder_target.shape}')\n",
    "print(f'test_encoder_input.shape = {test_encoder_input.shape},'\n",
    "      f'test_decoder_target.shape = {test_decoder_target.shape}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the IPU system\n",
    "cfg = ipu.utils.create_ipu_config()\n",
    "cfg = ipu.utils.auto_select_ipus(cfg, 1)\n",
    "ipu.utils.configure_ipu_system(cfg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "15918/15918 [==============================] - 160s 10ms/step - loss: 1.0263 - accuracy: 0.4597\n",
      "Epoch 2/5\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.9498 - accuracy: 0.5106\n",
      "Epoch 3/5\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.9165 - accuracy: 0.5317\n",
      "Epoch 4/5\n",
      "15918/15918 [==============================] - 63s 4ms/step - loss: 0.9037 - accuracy: 0.5401\n",
      "Epoch 5/5\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8952 - accuracy: 0.5454\n",
      "Epoch = 5,Validation Results = [0.9545077035637198, 0.52242774]\n",
      "Epoch 6/10\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.8897 - accuracy: 0.5486\n",
      "Epoch 7/10\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8836 - accuracy: 0.5523\n",
      "Epoch 8/10\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8793 - accuracy: 0.5564\n",
      "Epoch 9/10\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.8759 - accuracy: 0.5597\n",
      "Epoch 10/10\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8715 - accuracy: 0.5644\n",
      "Epoch = 10,Validation Results = [0.976505175471643, 0.5088124]\n",
      "Epoch 11/15\n",
      "15918/15918 [==============================] - 155s 10ms/step - loss: 0.8680 - accuracy: 0.5706\n",
      "Epoch 12/15\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8627 - accuracy: 0.5838\n",
      "Epoch 13/15\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.8463 - accuracy: 0.6061\n",
      "Epoch 14/15\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.8312 - accuracy: 0.6207\n",
      "Epoch 15/15\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.8200 - accuracy: 0.6303\n",
      "Epoch = 15,Validation Results = [0.8672440268258479, 0.58927506]\n",
      "Epoch 16/20\n",
      "15918/15918 [==============================] - 160s 10ms/step - loss: 0.8101 - accuracy: 0.6390\n",
      "Epoch 17/20\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.7981 - accuracy: 0.6488\n",
      "Epoch 18/20\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.7891 - accuracy: 0.6552\n",
      "Epoch 19/20\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.7778 - accuracy: 0.6620\n",
      "Epoch 20/20\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7692 - accuracy: 0.6669\n",
      "Epoch = 20,Validation Results = [0.8600842606684733, 0.59773797]\n",
      "Epoch 21/25\n",
      "15918/15918 [==============================] - 157s 10ms/step - loss: 0.7617 - accuracy: 0.6711\n",
      "Epoch 22/25\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7547 - accuracy: 0.6752\n",
      "Epoch 23/25\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7478 - accuracy: 0.6792\n",
      "Epoch 24/25\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.7426 - accuracy: 0.6823\n",
      "Epoch 25/25\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7368 - accuracy: 0.6851\n",
      "Epoch = 25,Validation Results = [0.9045551112726298, 0.5712182]\n",
      "Epoch 26/30\n",
      "15918/15918 [==============================] - 160s 10ms/step - loss: 0.7323 - accuracy: 0.6877\n",
      "Epoch 27/30\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7286 - accuracy: 0.6902\n",
      "Epoch 28/30\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7223 - accuracy: 0.6934\n",
      "Epoch 29/30\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.7198 - accuracy: 0.6948\n",
      "Epoch 30/30\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7151 - accuracy: 0.6976\n",
      "Epoch = 30,Validation Results = [0.8843042836069565, 0.60440624]\n",
      "Epoch 31/35\n",
      "15918/15918 [==============================] - 159s 10ms/step - loss: 0.7113 - accuracy: 0.6991\n",
      "Epoch 32/35\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.7110 - accuracy: 0.6996\n",
      "Epoch 33/35\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7047 - accuracy: 0.7030\n",
      "Epoch 34/35\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.7040 - accuracy: 0.7039\n",
      "Epoch 35/35\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.7001 - accuracy: 0.7060\n",
      "Epoch = 35,Validation Results = [0.8433260305088565, 0.6244227]\n",
      "Epoch 36/40\n",
      "15918/15918 [==============================] - 160s 10ms/step - loss: 0.6989 - accuracy: 0.7065\n",
      "Epoch 37/40\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6953 - accuracy: 0.7085\n",
      "Epoch 38/40\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6929 - accuracy: 0.7096\n",
      "Epoch 39/40\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.6912 - accuracy: 0.7106\n",
      "Epoch 40/40\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6883 - accuracy: 0.7124\n",
      "Epoch = 40,Validation Results = [0.8123022816730578, 0.63570136]\n",
      "Epoch 41/45\n",
      "15918/15918 [==============================] - 161s 10ms/step - loss: 0.6866 - accuracy: 0.7130\n",
      "Epoch 42/45\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6844 - accuracy: 0.7148\n",
      "Epoch 43/45\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6814 - accuracy: 0.7158\n",
      "Epoch 44/45\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.6802 - accuracy: 0.7166\n",
      "Epoch 45/45\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.6785 - accuracy: 0.7175\n",
      "Epoch = 45,Validation Results = [0.804108572331179, 0.64540136]\n",
      "Epoch 46/50\n",
      "15918/15918 [==============================] - 158s 10ms/step - loss: 0.6758 - accuracy: 0.7191\n",
      "Epoch 47/50\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6744 - accuracy: 0.7200\n",
      "Epoch 48/50\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6724 - accuracy: 0.7204\n",
      "Epoch 49/50\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.6706 - accuracy: 0.7213\n",
      "Epoch 50/50\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6688 - accuracy: 0.7228\n",
      "Epoch = 50,Validation Results = [0.7948372080434082, 0.65190464]\n",
      "Epoch 51/55\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.6662 - accuracy: 0.7242\n",
      "Epoch 52/55\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6653 - accuracy: 0.7244\n",
      "Epoch 53/55\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6644 - accuracy: 0.7251\n",
      "Epoch 54/55\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.6613 - accuracy: 0.7268\n",
      "Epoch 55/55\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6595 - accuracy: 0.7273\n",
      "Epoch = 55,Validation Results = [0.7918802357161434, 0.6554391]\n",
      "Epoch 56/60\n",
      "15918/15918 [==============================] - 159s 10ms/step - loss: 0.6589 - accuracy: 0.7281\n",
      "Epoch 57/60\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.6563 - accuracy: 0.7293\n",
      "Epoch 58/60\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6561 - accuracy: 0.7293\n",
      "Epoch 59/60\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.6533 - accuracy: 0.7311\n",
      "Epoch 60/60\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6524 - accuracy: 0.7313\n",
      "Epoch = 60,Validation Results = [0.8002902277152221, 0.6538289]\n",
      "Epoch 61/65\n",
      "15918/15918 [==============================] - 154s 10ms/step - loss: 0.6502 - accuracy: 0.7324\n",
      "Epoch 62/65\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6496 - accuracy: 0.7326\n",
      "Epoch 63/65\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6485 - accuracy: 0.7332\n",
      "Epoch 64/65\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.6465 - accuracy: 0.7342\n",
      "Epoch 65/65\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.6448 - accuracy: 0.7349\n",
      "Epoch = 65,Validation Results = [0.774638289842606, 0.6627749]\n",
      "Epoch 66/70\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.6442 - accuracy: 0.7355\n",
      "Epoch 67/70\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6416 - accuracy: 0.7363\n",
      "Epoch 68/70\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.6409 - accuracy: 0.7368\n",
      "Epoch 69/70\n",
      "15918/15918 [==============================] - 65s 4ms/step - loss: 0.6398 - accuracy: 0.7375\n",
      "Epoch 70/70\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6377 - accuracy: 0.7385\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch = 70,Validation Results = [0.795748585060899, 0.6550856]\n",
      "Epoch 71/75\n",
      "15918/15918 [==============================] - 157s 10ms/step - loss: 0.6362 - accuracy: 0.7392\n",
      "Epoch 72/75\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6358 - accuracy: 0.7390\n",
      "Epoch 73/75\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6346 - accuracy: 0.7399\n",
      "Epoch 74/75\n",
      "15918/15918 [==============================] - 63s 4ms/step - loss: 0.6332 - accuracy: 0.7405\n",
      "Epoch 75/75\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6313 - accuracy: 0.7412\n",
      "Epoch = 75,Validation Results = [0.8060504137931911, 0.6557925]\n",
      "Epoch 76/80\n",
      "15918/15918 [==============================] - 159s 10ms/step - loss: 0.6295 - accuracy: 0.7422\n",
      "Epoch 77/80\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.6298 - accuracy: 0.7421\n",
      "Epoch 78/80\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6273 - accuracy: 0.7428\n",
      "Epoch 79/80\n",
      "15918/15918 [==============================] - 63s 4ms/step - loss: 0.6268 - accuracy: 0.7434\n",
      "Epoch 80/80\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6253 - accuracy: 0.7440\n",
      "Epoch = 80,Validation Results = [0.7768711376176843, 0.66654885]\n",
      "Epoch 81/85\n",
      "15918/15918 [==============================] - 159s 10ms/step - loss: 0.6240 - accuracy: 0.7448\n",
      "Epoch 82/85\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6228 - accuracy: 0.7454\n",
      "Epoch 83/85\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6219 - accuracy: 0.7455\n",
      "Epoch 84/85\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.6202 - accuracy: 0.7465\n",
      "Epoch 85/85\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.6196 - accuracy: 0.7467\n",
      "Epoch = 85,Validation Results = [0.7925583433881846, 0.663254]\n",
      "Epoch 86/90\n",
      "15918/15918 [==============================] - 158s 10ms/step - loss: 0.6176 - accuracy: 0.7477\n",
      "Epoch 87/90\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6170 - accuracy: 0.7479\n",
      "Epoch 88/90\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.6159 - accuracy: 0.7484\n",
      "Epoch 89/90\n",
      "15918/15918 [==============================] - 62s 4ms/step - loss: 0.6153 - accuracy: 0.7484\n",
      "Epoch 90/90\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.6143 - accuracy: 0.7490\n",
      "Epoch = 90,Validation Results = [0.7913102899802988, 0.66411406]\n",
      "Epoch 91/95\n",
      "15918/15918 [==============================] - 157s 10ms/step - loss: 0.6123 - accuracy: 0.7501\n",
      "Epoch 92/95\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6116 - accuracy: 0.7499\n",
      "Epoch 93/95\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6096 - accuracy: 0.7514\n",
      "Epoch 94/95\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.6097 - accuracy: 0.7508\n",
      "Epoch 95/95\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6085 - accuracy: 0.7514\n",
      "Epoch = 95,Validation Results = [0.7974360655045808, 0.66357994]\n",
      "Epoch 96/100\n",
      "15918/15918 [==============================] - 158s 10ms/step - loss: 0.6068 - accuracy: 0.7522\n",
      "Epoch 97/100\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.6060 - accuracy: 0.7525\n",
      "Epoch 98/100\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6058 - accuracy: 0.7524\n",
      "Epoch 99/100\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.6048 - accuracy: 0.7534\n",
      "Epoch 100/100\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.6032 - accuracy: 0.7539\n",
      "Epoch = 100,Validation Results = [0.7854573514793528, 0.6663525]\n",
      "Epoch 101/105\n",
      "15918/15918 [==============================] - 158s 10ms/step - loss: 0.6021 - accuracy: 0.7541\n",
      "Epoch 102/105\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.6012 - accuracy: 0.7546\n",
      "Epoch 103/105\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5999 - accuracy: 0.7556\n",
      "Epoch 104/105\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.5997 - accuracy: 0.7553\n",
      "Epoch 105/105\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5976 - accuracy: 0.7561\n",
      "Epoch = 105,Validation Results = [0.8489385751297536, 0.6515669]\n",
      "Epoch 106/110\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.5970 - accuracy: 0.7565\n",
      "Epoch 107/110\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5956 - accuracy: 0.7576\n",
      "Epoch 108/110\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.5958 - accuracy: 0.7569\n",
      "Epoch 109/110\n",
      "15918/15918 [==============================] - 63s 4ms/step - loss: 0.5953 - accuracy: 0.7576\n",
      "Epoch 110/110\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5929 - accuracy: 0.7582\n",
      "Epoch = 110,Validation Results = [0.8162721376427213, 0.6583294]\n",
      "Epoch 111/115\n",
      "15918/15918 [==============================] - 157s 10ms/step - loss: 0.5924 - accuracy: 0.7584\n",
      "Epoch 112/115\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5929 - accuracy: 0.7579\n",
      "Epoch 113/115\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.5905 - accuracy: 0.7594\n",
      "Epoch 114/115\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.5901 - accuracy: 0.7595\n",
      "Epoch 115/115\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.5888 - accuracy: 0.7600\n",
      "Epoch = 115,Validation Results = [0.8197788798826373, 0.6597]\n",
      "Epoch 116/120\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.5887 - accuracy: 0.7602\n",
      "Epoch 117/120\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5875 - accuracy: 0.7608\n",
      "Epoch 118/120\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.5869 - accuracy: 0.7606\n",
      "Epoch 119/120\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.5863 - accuracy: 0.7609\n",
      "Epoch 120/120\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5845 - accuracy: 0.7621\n",
      "Epoch = 120,Validation Results = [0.8693055606502313, 0.6508836]\n",
      "Epoch 121/125\n",
      "15918/15918 [==============================] - 163s 10ms/step - loss: 0.5842 - accuracy: 0.7617\n",
      "Epoch 122/125\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.5844 - accuracy: 0.7618\n",
      "Epoch 123/125\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5832 - accuracy: 0.7622\n",
      "Epoch 124/125\n",
      "15918/15918 [==============================] - 63s 4ms/step - loss: 0.5821 - accuracy: 0.7630\n",
      "Epoch 125/125\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5813 - accuracy: 0.7629\n",
      "Epoch = 125,Validation Results = [0.8476925569422896, 0.6520225]\n",
      "Epoch 126/130\n",
      "15918/15918 [==============================] - 163s 10ms/step - loss: 0.5801 - accuracy: 0.7637\n",
      "Epoch 127/130\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5797 - accuracy: 0.7636\n",
      "Epoch 128/130\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5771 - accuracy: 0.7645\n",
      "Epoch 129/130\n",
      "15918/15918 [==============================] - 64s 4ms/step - loss: 0.5799 - accuracy: 0.7635\n",
      "Epoch 130/130\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5775 - accuracy: 0.7647\n",
      "Epoch = 130,Validation Results = [0.8173149922290484, 0.6627906]\n",
      "Epoch 131/135\n",
      "15918/15918 [==============================] - 156s 10ms/step - loss: 0.5768 - accuracy: 0.7651\n",
      "Epoch 132/135\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5762 - accuracy: 0.7654\n",
      "Epoch 133/135\n",
      "15918/15918 [==============================] - 56s 4ms/step - loss: 0.5744 - accuracy: 0.7662\n",
      "Epoch 134/135\n",
      "15918/15918 [==============================] - 66s 4ms/step - loss: 0.5742 - accuracy: 0.7664\n",
      "Epoch 135/135\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5730 - accuracy: 0.7660\n",
      "Epoch = 135,Validation Results = [0.8314708434484204, 0.6589224]\n",
      "Epoch 136/140\n",
      "15918/15918 [==============================] - 161s 10ms/step - loss: 0.5726 - accuracy: 0.7668\n",
      "Epoch 137/140\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5718 - accuracy: 0.7669\n",
      "Epoch 138/140\n",
      "15918/15918 [==============================] - 57s 4ms/step - loss: 0.5721 - accuracy: 0.7666\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 139/140\n",
      "15918/15918 [==============================] - 68s 4ms/step - loss: 0.5700 - accuracy: 0.7679\n",
      "Epoch 140/140\n",
      "15918/15918 [==============================] - 60s 4ms/step - loss: 0.5695 - accuracy: 0.7677\n",
      "Epoch = 140,Validation Results = [0.8489038099994473, 0.6518183]\n",
      "Epoch 141/145\n",
      "15918/15918 [==============================] - 158s 10ms/step - loss: 0.5687 - accuracy: 0.7683\n",
      "Epoch 142/145\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5688 - accuracy: 0.7683\n",
      "Epoch 143/145\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5672 - accuracy: 0.7683\n",
      "Epoch 144/145\n",
      "15918/15918 [==============================] - 68s 4ms/step - loss: 0.5680 - accuracy: 0.7684\n",
      "Epoch 145/145\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5667 - accuracy: 0.7690\n",
      "Epoch = 145,Validation Results = [0.8444097706455704, 0.6596175]\n",
      "Epoch 146/150\n",
      "15918/15918 [==============================] - 160s 10ms/step - loss: 0.5661 - accuracy: 0.7691\n",
      "Epoch 147/150\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5665 - accuracy: 0.7689\n",
      "Epoch 148/150\n",
      "15918/15918 [==============================] - 58s 4ms/step - loss: 0.5622 - accuracy: 0.7712\n",
      "Epoch 149/150\n",
      "15918/15918 [==============================] - 68s 4ms/step - loss: 0.5654 - accuracy: 0.7696\n",
      "Epoch 150/150\n",
      "15918/15918 [==============================] - 59s 4ms/step - loss: 0.5626 - accuracy: 0.7707\n",
      "Epoch = 150,Validation Results = [0.8388209878867875, 0.6588556]\n"
     ]
    }
   ],
   "source": [
    "strategy = ipu.ipu_strategy.IPUStrategy()\n",
    "all_results = [[1000, 0]]\n",
    "split_train_val = int(np.floor(len(train_encoder_input) * 0.8))\n",
    "\n",
    "with strategy.scope():\n",
    "    # Create an instance of the model\n",
    "    model = get_model_attention(n_hidden)\n",
    "\n",
    "    # Get the dataset\n",
    "    train_ds = create_dataset(train_encoder_input[:split_train_val], train_decoder_input[:split_train_val], \n",
    "                              train_decoder_target[:split_train_val], batch_size, method='train', shuffle=SHUFFLE)\n",
    "    val_ds = create_dataset(train_encoder_input[split_train_val:], train_decoder_input[split_train_val:], \n",
    "                            train_decoder_target[split_train_val:], batch_size, method='val')\n",
    "    test_ds = create_dataset(test_encoder_input, test_decoder_input, \n",
    "                             test_decoder_target, batch_size, method='prediction')\n",
    "\n",
    "    # Train the model\n",
    "    adam = keras.optimizers.Adam(lr=0.00001, beta_1=0.9, beta_2=0.999)\n",
    "    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=adam)\n",
    "    epoch_ = 0\n",
    "    epochs_per_fit = 5\n",
    "    \n",
    "    while epoch_ < epochs:\n",
    "        \n",
    "        model.fit(train_ds, steps_per_epoch=len(train_encoder_input) // batch_size,\n",
    "                  initial_epoch=epoch_, epochs=epoch_ + epochs_per_fit)\n",
    "        epoch_ = epoch_ + epochs_per_fit\n",
    "        result = model.evaluate(val_ds)\n",
    "        all_results.append(result)\n",
    "        print(f'Epoch = {epoch_},' f'Validation Results = {result}')\n",
    "\n",
    "        if all_results[-1][0] < all_results[-2][0]:\n",
    "            model.save_weights(saved_model_path)\n",
    "\n",
    "    model.load_weights(saved_model_path)\n",
    "    pred = model.predict(test_ds)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction horizon = 0\n",
      "accuracy_score = 0.8188926155257424\n",
      "classification_report =               precision    recall  f1-score   support\n",
      "\n",
      "           0     0.6942    0.5387    0.6067     21147\n",
      "           1     0.8550    0.9355    0.8935     98622\n",
      "           2     0.6972    0.5368    0.6065     19767\n",
      "\n",
      "    accuracy                         0.8189    139536\n",
      "   macro avg     0.7488    0.6703    0.7022    139536\n",
      "weighted avg     0.8083    0.8189    0.8093    139536\n",
      "\n",
      "-------------------------------\n",
      "Prediction horizon = 1\n",
      "accuracy_score = 0.7284356725146199\n",
      "classification_report =               precision    recall  f1-score   support\n",
      "\n",
      "           0     0.6243    0.4631    0.5317     27448\n",
      "           1     0.7759    0.8881    0.8282     86603\n",
      "           2     0.5997    0.4718    0.5281     25485\n",
      "\n",
      "    accuracy                         0.7284    139536\n",
      "   macro avg     0.6666    0.6076    0.6293    139536\n",
      "weighted avg     0.7139    0.7284    0.7151    139536\n",
      "\n",
      "-------------------------------\n",
      "Prediction horizon = 2\n",
      "accuracy_score = 0.7386695906432749\n",
      "classification_report =               precision    recall  f1-score   support\n",
      "\n",
      "           0     0.6615    0.5846    0.6207     31915\n",
      "           1     0.7935    0.8618    0.8263     78305\n",
      "           2     0.6441    0.5773    0.6089     29316\n",
      "\n",
      "    accuracy                         0.7387    139536\n",
      "   macro avg     0.6997    0.6746    0.6853    139536\n",
      "weighted avg     0.7319    0.7387    0.7336    139536\n",
      "\n",
      "-------------------------------\n",
      "Prediction horizon = 3\n",
      "accuracy_score = 0.744868707717005\n",
      "classification_report =               precision    recall  f1-score   support\n",
      "\n",
      "           0     0.6660    0.7179    0.6910     38439\n",
      "           1     0.8155    0.8215    0.8185     65998\n",
      "           2     0.6996    0.6303    0.6631     35099\n",
      "\n",
      "    accuracy                         0.7449    139536\n",
      "   macro avg     0.7271    0.7232    0.7242    139536\n",
      "weighted avg     0.7452    0.7449    0.7443    139536\n",
      "\n",
      "-------------------------------\n",
      "Prediction horizon = 4\n",
      "accuracy_score = 0.7460870313037495\n",
      "classification_report =               precision    recall  f1-score   support\n",
      "\n",
      "           0     0.7208    0.7507    0.7354     47952\n",
      "           1     0.8291    0.7615    0.7939     48048\n",
      "           2     0.6933    0.7240    0.7083     43536\n",
      "\n",
      "    accuracy                         0.7461    139536\n",
      "   macro avg     0.7477    0.7454    0.7459    139536\n",
      "weighted avg     0.7495    0.7461    0.7471    139536\n",
      "\n",
      "-------------------------------\n"
     ]
    }
   ],
   "source": [
    "evaluation_metrics(test_decoder_target, pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
